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Developmental  Test  and  Evaluation  Role  in  Assessing 

System  Reliability 

Christopher  DiPetto 

Deputy  Director, 

Developmental  Test  and  Evaluation,  Washington,  D.C. 

In  April  2007,  the  Under  Secretary  of  Defense  for  Acquisition,  Technology,  and  Logistics 
[USD  (AT^^L)]  requested  the  Chairman  of  the  Defense  Science  Board  (DSB)  to  establish  a 
task  force  on  developmental  testing.  A  subject  of  concern  was  that  although  several  initiatives 
had  been  implemented  within  the  Department  to  stress  better  system  engineering  practices, 
recent  history  shows  that  poor  peformance  during  initial  operational  test  and  evaluation 
(lOT^E)  “suggests  deficiencies  in  developmental  test  evaluation  (DT^E)  processes.” 

One  specific  problem  identified  was  the  lack  of  focus  on  reliability,  availability,  and 
maintainability  (RAM)  during  system  design.  The  Under  Secretary  requested  the  DSB 
address  this  problem  and  develop  recommendations  to  improve  the  Department’s  DT^E 
processes.  The  goal  is  to  identify  suitability  problems  early  enough  to  change  the  system  design 
while  in  development  versus  retrofitting  it  after  it  has  pe  formed  poorly  in  lOT^^E. 


The  Defense  Science  Board  (DSB)  Task 
Force  finished  their  study  in  early 
2008.  The  DSB’s  findings  concluded 
that  systemic  changes  to  acquisition 
processes  and  a  lack  of  a  disciplined 
systems  engineering  process  have  resulted  in  the  high 
failure  rates  in  suitability.  In  February 
2008,  the  Deputy  Under  Secretary  of 
Defense  (Acquisition  and  Technology) 
and  the  Director,  Operational  Test  and 
Evaluation  (DOTScE)  established  the 
Reliability  Improvement  Working  Group 
to  implement  three  specific  recommen¬ 
dations  of  the  DSB:  (a)  ensure  programs 
are  structured  with  a  viable  systems 
engineering  strategy  to  include  a  reliabil¬ 
ity,  availability,  and  maintainability 
(RAM)  growth  program  as  an  integral 
part  of  design  and  development,  (b) 
reconstitute  a  cadre  of  personnel  within 
the  Department  and  the  Services  with 
training  and  experience  in  test  and  evaluation  (T&E) 
and  RAM,  and  (c)  implement  the  Office  of  the 
Secretary  of  Defense  (OSD)  policy  to  integrate 
developmental  and  operational  testing. 

In  July  2008,  the  RIWG  recommended,  and  the 
USD  (AT&L)  issued,  a  new  RAM  policy  to  ensure 
RAM  requirements  are  incorporated  into  development 
contracts  and  system  designs,  and  evaluated  in  each 
phase  of  the  acquisition  life  cycle  (available  at  http;// 


www.acq.osd.mil/sse/dte/docs/USD-ATLMemo-RAM- 
Policy-21Jul08.pdf).  In  addition  to  OSD  policy,  there  are 
several  initiatives  within  the  Services  to  address  RAM 
during  the  development  phases  of  acquisition.  In  late 
2007,  the  Army  published  a  new  policy  mandating 
programs  establish  a  reliability  threshold  before  entrance 
into  Milestone  B.  This  new  policy  requires 
the  threshold  be  incorporated  into  the 
system  design  and  development  (SDD) 
contract.  Additionally,  the  system  is  ex¬ 
pected  to  meet  or  exceed  the  threshold 
value  for  reliability  by  the  conclusion  of  the 
first  system-level  test  in  SDD.  The  other 
Services  are  also  assuring  the  proper 
policies  are  in  place  to  focus  on  reliability 
during  system  development.  Currently, 
both  the  Air  Eorce  and  Navy  require  the 
system  developer  to  address  the  require¬ 
ments  for  reliability  during  system  design 
as  part  of  the  SDD  contract. 

Another  area  addressed  was  the  re¬ 
duction  in  personnel  with  experience  in  T&E  and 
RAM  backgrounds.  In  the  late  1990s,  Congress 
directed  several  cuts  to  the  military’s  acquisition 
workforce.  These  reductions,  according  to  the  DSB 
report,  “put  the  DoD  acquisition  workforce  on  a 
precipitous  path”  to  losing  vital  technical  expertise, 
while  at  the  same  time,  our  weapon  systems  are 
becoming  more  complex.  As  one  DSB  member  put  it: 
‘W^e  went  from  Insight,  to  Oversight  to  Out  of  Sight.” 


Mr.  Christopher  DiPetto 


29(3)  •  September  2008  223 


The  ITEA  Journal  of  Test  and  Evaluation  jite-29-03-06.3d  11/8/08  11:23:50  223 


DiPetto 


The  Services  are  reassessing  their  acquisition  man¬ 
power  allocations  to  ensure  there  is  the  proper  focus  on 
growing  the  experience  levels  of  the  T&E  and  RAM 
workforce.  Additionally,  the  Defense  Acquisition 
University  has  taken  the  initiative  to  examine  their 
course  curriculums  to  ensure  our  workforce  is  properly 
trained  to  employ  sound  T&E  and  RAM  principles 
during  system  development. 

The  final  focus  area  is  the  implementation  of  an 
integrated  test  policy  within  the  Department.  Inte¬ 
grated  testing  is  the  process  of  collaborative  planning 
and  execution  of  test  phases  and  events  to  ensure  the 
objectives  of  the  stakeholders  (both  operational  and 
developmental)  are  addressed.  One  of  the  primary 
purposes  of  T&E  is  to  ensure  that  the  system,  as 
designed,  will  meet  the  warfighter’s  requirements  in 
the  operational  environment.  Operational  Test  and 
Evaluation  (OT&E)  evaluates  the  operational  effec¬ 
tiveness  and  operational  suitability  of  the  design.  By 
the  time  a  system  is  ready  for  OT&E  the  design  is 
pretty  much  fixed — it’s  too  late  to  make  major  changes. 
One  of  the  fundamental  focuses  of  developmental  test 
&  evaluation  (DT&E)  is  to  test  and  evaluate  the  system 
design  to  ensure  it  will  meet  the  warfighter’s  require¬ 
ments.  System  developers  and  the  DT&E  community 
use  the  Joint  Staff  validated  capability  requirements 
documents  as  the  source  for  system  requirements. 
However,  as  is  often  the  case,  the  Concepts  of 
Operation  (CONOPs)  is  not  made  available  (or  is 
not  used)  during  SDD  to  help  define  the  scope  of 
developmental  tests.  Therefore,  the  system  is  not 
properly  stressed  during  SDD  when  there  is  still  time 


to  make  design  changes.  Integrated  testing  brings  a 
“mission-oriented”  approach  during  DT&E  by  getting 
all  the  team  players  involved  in  system  development 
(contractor,  program  office,  user,  developmental  test, 
and  operational  test)  to  incorporate  the  mission  context 
into  the  developmental  test  strategy.  That  way,  the 
system  design  can  be  tested  based  on  how  the  system 
will  be  employed.  Not  taking  the  operational  environ¬ 
ment  into  account  during  development  is  akin  to  an 
automobile  manufacturer  building  a  half-ton  pickup 
not  knowing  that  the  user  needs  a  four-wheel  drive 
truck.  Both  vehicles  carry  the  required  tonnage,  but 
how  it  will  be  used  (off-road)  was  not  taken 
into  account  when  the  truck  was  on  the  drawing 
board.  By  using  the  CONOPs  during  the  develop¬ 
ment  of  the  statement  of  work,  the  system  developer 
and  manufacturer  would  have  known  the  truck 
would  be  employed  on  rough  terrain  and  would 
have  incorporated  a  robust  suspension  system,  four- 
wheel  drive,  etc.  into  the  design.  The  vehicle  would 
have  then  been  tested  in  an  off-road  environment 
before  being  delivered  to  the  customer.  Although 
much  more  complicated,  the  same  principle  applies  to 
weapon  systems.  Additionally,  by  infusing  a  mission- 
oriented  approach  during  DT&E,  data  that  is 
operationally  representative  can  be  used  to  reduce  the 
scope  of  initial  operational  test  and  evaluation. 
Integrated  Testing  should  not  only  save  time  and 
money  in  the  test  program,  but  the  real  savings  will  be 
in  the  dollars  and  time  saved  by  less  redesigns  and 
retrofits  after  the  system  is  in  production.  It’s  a  “win- 
win”  proposition.  □ 
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